Head-mounted type display device and method of controlling head-mounted type display device

ABSTRACT

An HMD includes an image display unit configured to display objects in a display region (VR) overlapping with outside scenery; a storage unit configured to store a process flow data to be used for selecting the objects to be displayed in the display region (VR); and a control unit configured to select any of the objects to be displayed in the display region (VR), based on the selection information, and to control display positions of the objects in a depth direction of the display region (VR) so that a display position of a selected object is different from a display position of another object to be displayed in the display region (VR).

BACKGROUND 1. Technical Field

The invention relates to a head-mounted type display device and a methodof controlling the head-mounted type display device.

2. Related Art

Hitherto, it has been known a head-mounted type display device capableof securing visibility of objects in a case of displaying the objectsincluding characters (for example, refer to JP-A-2014-95903).

In an image display method of a head-mounted type display inJP-A-2014-95903, chromaticity of an image to be displayed is set so thatchromaticity of a predetermined region including an observation objectviewed through an image display device and the chromaticity of the imageto be displayed establish approximate complementary color relationship.

The head-mounted type display device has been considered to be used invarious applications. Thus, there is a need to effectively utilize thehead-mounted type display device in various applications by effectivelyutilizing information displayed by the head-mounted type display device.However, even when the head-mounted display device displays theinformation, a user may miss the displayed information, and thus thisleads a problem that the displayed information may not effectively beutilized.

SUMMARY

The present invention has an object to reduce an oversight ofinformation and effectively utilize the information displayed by ahead-mounted type display device.

In order to achieve the above-mentioned object, an aspect of the presentinvention is a head-mounted type display device configured to be mountedon a head of a user and through which outside scenery is visible, thehead-mounted type display device including a display unit configured todisplay objects in a display region overlapping with the outsidescenery, a storage unit configured to store selection information to beused for selecting the objects to be displayed in the display region,and a control unit configured to select any of the objects to bedisplayed in the display region, based on the selection information, andto control display positions of the objects in a depth direction of thedisplay region so that a display position of a selected object isdifferent from a display position of another object to be displayed inthe display region.

With this configuration, the selected object can be displayed at adifferent display position from that of another object in the depthdirection. Thus, an oversight of the objects can be reduced, and theinformation displayed by the head-mounted type display device caneffectively be utilized.

In an aspect of the present invention, the control unit is configured toadjust the display positions of the objects in the depth direction bychanging a convergence angle between a right eye and a left eye of theuser.

With this configuration, the display positions of the objects in thedepth direction can easily be adjusted.

In an aspect of the present invention, the control unit is configured tocontrol the display positions of the objects in the depth direction ofthe display region and display states of the objects.

With this configuration, the display positions and the display states ofthe objects are controlled. Accordingly, an oversight of the objects canfurther be reduced, and the information displayed by the head-mountedtype display device can effectively be utilized.

In an aspect of the present invention, the control unit is configured tochange the display position of the selected object in the depthdirection every unit time.

With this configuration, the display position of the selected object inthe depth direction can be changed every unit time. Accordingly, anoversight of the objects can further be reduced, and the informationdisplayed by the head-mounted type display device can effectively beutilized.

In an aspect of the present invention, the control unit is configured tochange the display position of the selected object in the depthdirection, and alternately switch a display state every unit timebetween a display state in which the selected object is displayed on auser side than another object in the depth direction and a display statein which the selected object is displayed on a far side than anotherobject in the depth direction.

With this configuration, the display position of the selected object inthe depth direction can be changed every unit time. Accordingly, anoversight of the objects can further be reduced, and the informationdisplayed by the head-mounted type display device can effectively beutilized.

In an aspect of the present invention, the control unit is configured tochange the display position of the selected object about an imaginaryaxis set in a predetermined direction with the display region as areference.

With this configuration, the display position of the selected object ischanged about the imaginary axis. Thus, an oversight of the objects canfurther be reduced, and the information displayed by the head-mountedtype display device can effectively be utilized.

In an aspect of the present invention, the control unit is configured toset the imaginary axis in a direction parallel with a vertical directionof the display region. The control unit is configured to alternatelyswitch a display state every unit time between a display state in whichone end of the selected object in a direction parallel with a horizontaldirection of the display region is displayed on a user side in the depthdirection than another end in the direction parallel with the horizontaldirection of the display region and a display state in which the otherend is displayed on a user side than the one end in the depth direction.

With this configuration, the display positions of the one end and theother end of the selected object in the depth direction can be changedevery unit time. Thus, an oversight of the objects can further bereduced, and the information displayed by the head-mounted type displaydevice can effectively be utilized.

In an aspect of the present invention, the control unit is configured toalternately switch a display state every unit time between a displaystate in which a length of the selected object on the one end side isshorter than a length of the selected object on the other end and adisplay state in which the length of the selected object on the otherend side is shorter than the length of the selected object on the oneend side.

With this configuration, the lengths of the one end and the other end ofthe selected object can be changed every unit time. Accordingly, anoversight of the objects can further be reduced, and the informationdisplayed by the head-mounted type display device can effectively beutilized.

In an aspect of the present invention, the control unit is configuredto, in a case where the selected object is a character string and theother object includes a character string, change the display state ofthe selected object so that at least any of a color, boldness, abackground color, a font type, and a font size of the character stringas the selected object and at least corresponding one of a color,boldness, a background color, a font type, and a font size of thecharacter string of the other object are different from each other.

With this configuration, at least any of a color, boldness, a backgroundcolor, a font type, and a font size of the character string as theselected object can be changed from that of the other character string.Therefore, the user recognizes the selected character string easily, andan oversight of the information needed for the user can further bereduced with the head-mounted type display device. Thus, the informationdisplayed by the head-mounted type display device can effectively beutilized.

In an aspect of the present invention, the storage unit is configured tostore control data defining an operation flow including a plurality ofoperation steps for which an execution order is set in advance. Thecontrol unit is configured to, in a case where the operation flow isexecuted, switch whether or not to execute control of the displaypositions of the objects in accordance with setting of the control dataregarding the plurality of operation steps.

With this configuration, the display positions of the objects in thedepth direction, which are included in the operation flow, can bechanged every unit time. Thus, the user can recognize the selectedobject more easily, and an oversight can further be reduced. Therefore,in a case of using the head-mounted type display device for worksupport, work of the user can effectively be supported.

In an aspect of the present invention, the control unit is configuredto, in a case where a plurality of the objects are selected, cause theselected plurality of objects to be sequentially displayed in thedisplay region.

With this configuration, the plurality of objects can be displayed in anorder of selection. Thus, even when the plurality of objects areselected, an oversight of the objects can be reduced, and theinformation displayed by the head-mounted type display device caneffectively be utilized.

In an aspect of the present invention, the control unit is configuredto, in a case where a first object and a second object are selected asthe plurality of objects, control the display positions of the objectsin the display region so that a display state is alternately switchedbetween a first display state in which the first object is displayed ona user side than another object and the second object is displayed on afar side than another object in the depth direction, and a seconddisplay state in which the second object is displayed on a user sidethan another object and the first object is displayed on a far side thananother object in the depth direction.

With this configuration, the display states of the first object and thesecond object can alternately be changed between the first display stateand the second display state. Thus, even when the plurality of objectsare selected, an oversight of the objects can be reduced, and theinformation displayed by the head-mounted type display device caneffectively be utilized.

In an aspect of the present invention, the control unit is configured tocontrol the display position of the selected object in the displayregion so that the selected object does not overlap with another object.

With this configuration, the plurality of object are displayed in thedisplay region not to overlap with each other. Accordingly, degradationof visibility of the objects can be prevented.

In an aspect of the present invention, the control unit is configured toselect the object including any of a character, an icon, and a framebody surrounding at least any of the character and the icon.

With this configuration, the display positions of the character, theicon, and the frame body surrounding at least any of the character andthe icon are controlled. Thus, an oversight of the objects can bereduced, and the information displayed by the head-mounted type displaydevice can effectively be utilized.

In order to achieve the above-mentioned object, an aspect of the presentinvention is a method of controlling a head-mounted type display deviceconfigured to be mounted on a head of a user and through which outsidescenery is visible, the method including selecting any of objects to bedisplayed in a display region overlapping with the outside scenery,based on selection information to be used for selecting the objects, andcontrolling display positions of the objects in a depth direction of thedisplay region so that a display position of a selected object isdifferent from a display position of another object to be displayed inthe display region.

With this configuration, the selected object can be displayed at adifferent display position from that of another object in the depthdirection. Thus, an oversight of the objects can be reduced, and theinformation displayed by the head-mounted type display device caneffectively be utilized.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a view illustrating an external configuration of an HMD.

FIG. 2 is a plan view illustrating a main part of a configuration of anoptical system of the HMD.

FIG. 3 is a perspective view illustrating a configuration of an imagedisplay unit.

FIG. 4 is a block diagram of the HMD.

FIG. 5 is a function block diagram of a control device.

FIG. 6 is a view illustrating a relationship between a size of areference region and a size of an emitting region.

FIG. 7 is an explanatory view for illustrating an image formed by imagelight.

FIG. 8 is a view illustrating a display example of a display region.

FIG. 9 is a view illustrating a display example of an emphasizedcharacter string displayed in the display region.

FIG. 10 is a view illustrating a display position of the characterstring in a Z axis direction.

FIG. 11 is a view illustrating a display position of the characterstring in the Z axis direction.

FIGS. 12A and 12B are views each illustrating a display position of thecharacter string in the Z axis direction.

FIGS. 13A to 13C are views each illustrating the emphasized characterstring displayed in the display region.

FIGS. 14A to 14C are views each illustrating the emphasized characterstring displayed in the display region.

FIGS. 15A to 15C are views each illustrating the emphasized characterstring displayed in the display region.

FIGS. 16A and 16B are views each illustrating the emphasized characterstring displayed in the display region.

FIG. 17 is a view illustrating display of the display region.

FIG. 18 is a flowchart illustrating an operation of a control unit.

FIG. 19 is a view illustrating display of the display region.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

An exemplary embodiment of the invention is described below withreference to the drawings.

FIG. 1 is an explanatory view for illustrating an external configurationof a Head Mounted Display (HMD) 100 to which the present invention isapplied. The HMD 100 is a display device including a control device 10and an image display unit 20. The image display unit 20 corresponds to a“display unit” of the present invention.

The control device 10 includes an operation unit configured to receivean operation by a user, and functions as a controller through which theuser operates the HMD 100. The control device 10 receives an operationby the user, and controls the image display unit 20 according to thereceived operation. The image display unit 20 is mounted on a user'shead, and allows the user to visually recognize a virtual image. Theuser refers to a person with the image display unit 20 mounted onhis/her head.

As illustrated in FIG. 1, the control device 10 includes a flat box-likecase 10Au, which can also be described as a hosing or a main body. Thecase 10A includes portions such as operation buttons 11, an LEDindicator 12, a track pad 14, up and down keys 15, a changeover switch16, and a power switch 18. The operation buttons 11, the up and downkeys 15, the changeover switch 16, and the power switch 18 arecollectively referred to as operation elements 13 (FIG. 4). The user canoperate the HMD 100 by operating the operation elements 13 and the trackpad 14.

The operation buttons 11 includes a menu key, a home key, and a returnkey, and particularly some of the keys and switches are transformed by apressing operation. The LED indicator 12 turns on and blinks accordingto the operation state of the HMD 100.

The track pad 14 includes an operation surface for detecting a contactoperation, and outputs an operation signal in response to the operationon the operation surface. As a detection system for detecting operationson the operation surface of the track pad 14, various detection systemssuch as an electrostatic type and an optical type can be employed.

The up and down keys 15 are used for instruction inputting of increasingor reducing volume of sound output from a right earphone 32 and a leftearphone 34 and for instruction inputting of increasing or reducingbrightness of display of the image display unit 20. The changeoverswitch 16 is a switch for changing over inputting corresponding tooperations of the up and down keys 15. The power switch 18 is a switchfor changing over an on-state and an off-state of power of the HMD 100,and is formed of, for example, a slide switch.

The image display unit 20 is a head-mounted body to be worn on theuser's head and has an eyeglasses-like shape in the exemplaryembodiment. The image display unit 20 includes a main body including aright holding part 21, a left holding part 23, and a front frame 27, andfurther includes, on the main body, a right display unit 22, a leftdisplay unit 24, a right light guide plate 26, and a left light guideplate 28.

The right holding part 21 and the left holding part 23 extend rearwardfrom respective ends of the front frame 27, to hold the image displayunit 20 on the user's head like temples of eyeglasses. Here, an end ERrefers to one of the opposite ends of the front frame 27 positioned onthe user's right side in a state where the image display unit 20 ismounted on the user. An end EL refers to the other end positioned on theuser's left side in the mounted state. The right holding portion 21 isprovided to extend from the end ER of the front frame 27 to a positioncorresponding to the right side of the user's head in the state wherethe image display unit 20 is mounted on the user. The left holdingportion 23 is provided to extend from the end EL to a positioncorresponding to the left side of the user's head in the state where theimage display unit 20 is mounted on the user.

The right light guiding plate 26 and the left light guiding plate 28 areprovided in the front frame 27. The right light guide plate 26 ispositioned in front of the user's right eye in the state where the imagedisplay unit 20 is mounted on the user, to allow the right eye tovisually recognize an image. The left light guide plate 28 is positionedin front of the user's left eye in the state where the image displayunit 20 is mounted on the user, to allow the right eye to visuallyrecognize an image.

The front frame 27 is shaped to couple one end of the right light guideplate 26 and one end of the left light guide plate 28 together, and thecoupling position between the right light guide plate 26 and the leftlight guide plate 28 corresponds to a position of the user's glabella inthe state where the image display unit 20 is mounted on the user. Thefront frame 27 may include a nose pad part provided at the couplingposition between the right light guide plate 26 and the left light guideplate 28, the nose pad part abutting on the user's nose in the statewhere the image display unit 20 is mounted on the user. In this case,the nose pad part, the right holding part 21, and the left holding part23 allow the image display unit 20 to be held on the user's head. A belt(not illustrated) may also be coupled to the right holding portion 21and the left holding portion 23 to contact the back of the user's headin the state where the image display unit 20 is mounted on the user. Inthis case, the belt allows the image display unit 20 to be held on theuser's head.

The right display unit 22 enables images to be displayed through theright light guiding plate 26. The right display unit 22 is provided onthe right holding part 21, and positioned near the right side region ofthe user's head in the state where the image display unit 20 is mountedon the user. The left display unit 24 enables images to be displayedthrough the left light guiding plate 28. The left display unit 24 isprovided on the left holding part 23, and positioned near the left sideregion of the user's head in the state where the image display unit 20is mounted on the user.

Each of the right light guiding plate 26 and the left light guidingplate 28 is an optical part made of light transmissive resin and thelike, and is formed of, for example, a prism. The right light guidingplate 26 and the left light guiding plate 28 guide image light outputfrom the right display unit 22 and the left display unit 24 to the eyesof the user.

Surfaces of the right light guiding plate 26 and the left light guidingplate 28 may be provided with dimmer plates (not illustrated). Thedimmer plates are optical elements on a thin plate, having a differenttransmittance for a different wavelength range of light, and function asso-called wavelength filters. The dimmer plates are, for example,arranged so as to cover a front side of the front frame 27, which is aside opposite to the side of the user's eyes. Appropriate selection ofoptical characteristics of the dimmer plates allows adjustment oftransmittance of light with an arbitrary wavelength region such asvisible light, infrared light, or ultraviolet light and allowsadjustment of the amount of outside light entering the right light guideplate 26 and the left light guide plate 28 from outside and passingthrough the right light guide plate 26 and the left light guide plate28.

The image display unit 20 is a transmissive display device for causingthe outside light to pass through the image display unit 20 to therebyallow outside scenery to be visually recognized. The image display unit20 guides the image light generated in the right display unit 22 and theimage light generated in the left display unit 24 to the right lightguiding plate 26 and the left light guiding plate 28, respectively. Theimage light guided by the right light guiding plate 26 and the imagelight guided by the left light guiding plate 28 enter the right eye andthe left eye of the user, respectively, to allow the user to visuallyrecognize a virtual image. In this manner, the image display unit 20displays an image. In a visual field range FV of the user wearing theimage display unit 20 on his/her head, a region in which the imagedisplay unit 20 can display an image is referred to as a display regionVR. The display region VR is superimposed on outside scenery, and theuser can visually recognize the object displayed by the image displayunit 20 and the outside scenery at the same time. The object will bedescribed later in detail.

In a case where the outside light travels from the front side of theuser passes through the right light guiding plate 26 and the left lightguiding plate 28 and enters the eyes of the user, image light forming avirtual image and the outside light enters the eyes of the user. Thus,visibility of a visual image is affected by intensity of the outsidelight. For this reason, the visibility of a visual image can beadjusted, for example, by installing the dimmer plates on the frontframe 27 and appropriately selecting or adjusting the opticalcharacteristics of the dimmer plates. In a typical example, the dimmerplate having a light transmissivity sufficient to allow the user wearingthe HMD 100 to visually recognize at least the outside scenery may beused. The use of the dimmer plates is also expected to be effective forprotecting the right light guide plate 26 and the left light guide plate28 to suppress, e.g., damage and adhesion of stains to the right lightguide plate 26 and the left light guide plate 28. The dimmer plates maybe detachable from the front frame 27 or each of the right light guidingplate 26 and the left light guiding plate 28. A plurality of types ofdimmer plates may be mountable to selectively install any of the dimmerplates by replacement. The image display unit 20 may be configuredwithout the dimmer plates.

A camera 60 is arranged on the front frame 27 of the image display unit20. The configuration and arrangement of the camera 60 is determined soas to capture an image of outside scenery to be visually recognized bythe user wearing the image display unit 20. The outside scenery refersto scene of outside in the direction of line of sight of the userwearing the image display unit 20 on the head. For example, the camerais provided in a front surface of the front frame 27 at a position wherethe camera 61 does not block the outside light transmitted through theright light guide plate 26 and the left light guide plate 28. In theexample illustrated in FIG. 1, the camera 60 is arranged on the end ERside of the front frame 27, and functions as a detection unit fordetecting a part of the body or an object attached on the body of theuser as an indicator.

The camera 60 is a digital camera equipped with an imaging lens, animaging element such as a CCD or a CMOS, and the like. In the exemplaryembodiment, the camera 60 is a monocular camera, but may be a stereocamera. The camera 60 captures at least part of outside scenery in thefront side direction of the HMD 100, in other words, in a direction ofthe field of view of the user wearing the HMD 100. In other words, thecamera 60 captures an image of a range overlapping with the field ofview of the user or the direction of the field of view of the user,i.e., in a direction of field of fixation by the user. A direction and awidth of an angle of view of the camera 60 may selectively be set. Inthe exemplary embodiment, as described later, the angle of view of thecamera 60 includes outside world visually recognized by the user throughthe right light guiding plate 26 and the left light guiding plate 28.More preferably, the angle of view of the camera 60 is set to so thatthe entire field of view that the user can visually recognize throughthe right light guiding plate 26 and the left light guiding plate 28 canbe captured.

The camera 60 performs image capturing in accordance with control of animaging control unit 153 (FIG. 5) provided with the control unit 150.When the power switch 18 is turned on and a main processor 140 receivespower supply from a power supply unit 130 to be activated, the mainprocessor 140 causes the power supply unit 130 to start power supply tothe camera 60, and causes the camera 60 to turn on. When the camera 60turns on, the imaging control unit 153 causes the camera 60 to performimage capturing every predetermined time period to generate capturedimage data. The captured image data is written in a memory 121 (refer toFIG. 4), and is temporarily stored.

The HMD 100 may include a distance sensor (not illustrated) fordetecting a distance to a measurement object positioned in a presetmeasurement direction. The distance sensor may be arranged at theconnection between the right light guiding plate 26 and the left lightguiding plate 28 of the front frame 27, for example. In this case, inthe state where the image display unit 20 is mounted on the user, theposition of the distance sensor is substantially in the middle of boththe eyes of the user in the horizontal direction and on an upper side ofboth the eyes of the user in the vertical direction. The measurementdirection of the distance sensor may be, for example, the frontdirection of the front frame 27, in other words, a direction overlappingwith an image capturing direction of the camera 60. The distance sensormay include, for example, a light source such as an LED or a laserdiode, and a light-receiving unit for receiving light that is emitted bythe light source and is reflected by the measurement object. Thedistance sensor is only required to perform measurement processing basedon triangular measurement processing or a time difference in accordancewith control of the control unit 150. The distance sensor may include,for example, a sound source for transmitting an ultrasonic wave and adetection unit for receiving the ultrasonic wave reflected by themeasurement object. In this case, the distance sensor is only requiredto perform measurement processing based on a time difference to thereflection of the ultrasonic wave in accordance with control of thecontrol unit 150.

FIG. 2 is a plan view illustrating a main part of a configuration of anoptical system included in the image display unit 20. For description,in FIG. 2, a left eye LE and a right eye RE of the user are illustrated.

As illustrated in FIG. 2, the right display unit 22 and the left displayunit 24 are configured laterally symmetrically. As a configurationallowing the right eye RE of the user to visually recognize an image,the right display unit 22 includes an Organic Light-Emitting Diode(OLED) unit 221 for emitting image light and a right optical system 251including a lens group for guiding image light L emitted by the OLEDunit 221. The image light L is guided to the right light guiding plate26 by the right optical system 251.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit225 configured to drive the OLED panel 223. The OLED panel 223 is aself-light-emitting display panel including light emitting elementsemitting red (R) color light, green (G) color light, and blue (B) colorlight by organic electro-luminescence are arranged in a matrix. The OLEDpanel 223 includes a plurality of pixels, each pixel corresponding to aunit including one element of R, one element of G, and one element of B.An image is formed by the pixels arranged in a matrix. In accordancewith the control of the control unit 150 (FIG. 5), the OLED drivecircuit 225 selects the light-emitting elements and applies a current tothe light-emitting elements included in the OLED panel 223 to cause thelight-emitting elements of the OLED panel 223 to emit light. The OLEDdrive circuit 225 is fixed by bonding or the like, to a back surface ofthe OLED panel 223, i.e., a back side of a light-emitting surface. TheOLED drive circuit 225 may include a semiconductor device for drivingthe OLED panel 223, for example, and may be mounted on a substrate (notillustrated) fixed to the back surface of the OLED panel 223. Atemperature sensor 217 is mounted on the substrate.

The OLED panel 223 may have a configuration in which light-emittingelements to emit white color light are arranged in a matrix and colorfilters corresponding to the R color, the G color, and the B color, arearranged to overlap with the light-emitting elements. Further, inaddition to the light-emitting elements emitting R color light, G colorlight, and B color light, the OLED panel 223 with a WRGB configurationincluding light-emitting elements emitting white (W) color light may beadopted.

The right optical system 251 includes a collimate lens configured tocollimate the image light L emitted from the OLED panel 223. The imagelight L collimated by the collimate lens enters the right light guidingplate 26. An optical path configured to guide light in the right lightguiding plate 26 is formed with a plurality of reflective facesconfigured to reflect the image light L. The image light L reflectedmultiple times in the right light guiding plate 26 is guided to theright eye RE. The right light guiding plate 26 is formed with a halfmirror 261 (reflective surface) in front of the right eye RE. The imagelight L is reflected by the half mirror 261, is emitted from the rightlight guiding plate 26 to the right eye RE. The image light L forms animage on the retina of the right eye RE to allow the user to visuallyrecognize the image.

Further, as a configuration for allowing the user to visually recognizean image with the left eye LE, the left display unit 24 includes an OLEDunit 241 for emitting image light and a left optical system 252including a lens group for guiding the image light L emitted by the OLEDunit 241. The image light L is guided to the left light guiding plate 28by the left optical system 252.

The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit245 for driving the OLED panel 243. The OLED panel 243 is a lightemission type display panel similar to the OLED panel 223. In accordancewith the control of the control unit 150 (FIG. 5), the OLED drivecircuit 245 selects the light-emitting elements and applies a current tothe light-emitting elements included in the OLED panel 243 to cause thelight-emitting elements of the OLED panel 243 to emit light. The OLEDdrive circuit 245 is fixed by bonding or the like, to a back surface ofthe OLED panel 243, i.e., a back side of a light-emitting surface. TheOLED drive circuit 245 may include a semiconductor device for drivingthe OLED panel 243, for example, and may be mounted on a substrate (notillustrated) fixed to the back surface of the OLED panel 243. Atemperature sensor 239 is mounted on the substrate.

The left optical system 252 includes a collimate lens for collimatingthe image light L emitted from the OLED panel 243. The image light Lcollimated by the collimate lens enters the left light guiding plate 28.The left light guiding plate 28 is an optical element having a pluralityof reflective surface configured to reflect the image light L, and isformed of, for example, a prism. The image light L reflected multipletimes in the left light guiding plate 28 is guided to the left eye LE.The left light guiding plate 28 is formed with a half mirror 281(reflective surface) in front of the right eye LE. The image light Lreflected by the half mirror 281 is emitted from the left light guidingplate 28 to the left eye LE. The image light L forms an image on theretina of the left eye LE to allow the user to visually recognize theimage.

With this configuration, the HMD 100 functions as a see-through typedisplay device. That is, the image light L reflected by the half mirror261 and the outside light OL passing through the right light guidingplate 26 enter the right eye RE of the user. The image light L reflectedby the half mirror 281 and the outside light OL passing through the halfmirror 281 enter the left eye LE. In this manner, the HMD 100 allows theimage light L of the internally processed image and the outside light OLto be combined to enter the user's eyes. The user sees outside scenerythrough the right light guiding plate 26 and the left light guidingplate 28, and an image formed by the image light L is displayed to bevisually recognizable in an overlapped manner with the outside sceneryor around the outside scenery. The half mirrors 261 and 281 are imagepickup units for reflecting the image light output from the rightdisplay unit 22 and the left display unit 24 and picking up the images,and may be referred to as display units.

Note that, the left optical system 252 and the left light guiding plate28 are collectively referred to as a “left light-guiding unit”. Theright optical system 251 and the right light guiding plate 26 arecollectively referred to as a “right light-guiding unit”. Theconfigurations of the right light-guiding unit and the leftlight-guiding unit are not limited to the example described above, andany scheme may be adopted as long as image light forms a visual image infront of the user's eyes. For example, a diffraction grating may beused, and a transflective film may be used.

Referring back to FIG. 1, the control device 10 and the image displayunit 20 are coupled via a coupling cable 40. The coupling cable 40 isdetachably coupled to a connector provided at a lower part of the case10A, and connects to various circuits inside the image display unit 20through a tip of the left holding portion 23. The coupling cable 40includes a metal cable or an optical fiber cable for transmittingdigital data, and may include a metal cable for transmitting analogsignals. A connector 46 is provided at a mid-point of the coupling cable40. The connector 46 is a jack to which a stereo mini-plug is connected,and is coupled to the control device 10, for example, via a line throughwhich analog voice signals are transmitted. In the configuration exampleillustrated in FIG. 1, the connector 46 is coupled to a headset 30including the right earphone 32 and the left earphone 34, which form astereo headphone, and a microphone 63.

The control device 10 and the image display unit 20 may be coupled in awireless manner. For example, the control device 10 and the imagedisplay unit 20 may be configured to mutually transmit and receivecontrol signals and data through wireless communication conformingstandards such as Bluetooth (registered trademark) and a wireless LAN(including Wi-Fi (registered trademark)).

As illustrated in FIG. 1, for example, the microphone 63 is arranged sothat a sound collector of the microphone 63 is oriented in the directionof the user's line of sight, collects sound, and outputs sound signalsto a sound interface 180 (FIG. 4). The microphone 63 may be, forexample, a monaural microphone or a stereo microphone, or may be adirectional microphone or a non-directional microphone.

FIG. 3 is a perspective view for illustrating a configuration of theimage display unit 20 and illustrates a main configuration when theimage display unit 20 is viewed from the user's head. In FIG. 3, theside of the image display unit 20 that come into contact with the user'shead, in other words, the side that can be seen by the user with theright eye RE and the left eye LE are illustrated. In other words, theback sides of the right light guiding plate 26 and the left lightguiding plate 28 are illustrated.

In FIG. 3, the half mirror 261 for emitting image light to the right eyeRE of the user and the half mirror 281 for emitting image light to theleft eye LE are also illustrated as approximately quadrangle-shapedregions. Further, as described above, outside light passes through theentire areas of the right light guiding plate 26 and the left lightguiding plate 28 including the half mirrors 261 and 281. Thus, the uservisually recognizes outside scenery through the entire areas of theright light guiding plate 26 and the left light guiding plate 28, andvisually recognizes rectangular display images at the positions of thehalf mirrors 261 and 281.

The camera 60 is arranged at an end on the right side of the imagedisplay unit 20 to capture an image in the direction of the line ofsight of both user's eyes, i.e., in front of the user. The optical axisof the camera 60 extends in a direction including the direction of theline of sight of the right eye RE and the left eye LE. The outsidescenery that can be visually recognized by the user wearing the HMD 100is not necessarily at infinity. For example, in a case where the usergazes a target object positioned in front of the user with both theeyes, the distance from the user to the object often ranges fromapproximately 30 cm to 10 m, and more often ranges from 1 m to 4 m.Based upon this, standard maximum and minimum distances from the user tothe object that the user can take during normal use of HMD 100 may bespecified. The standards may be obtained through investigation andexperiments, and may be set by the user. It is preferred that theoptical axis and the angle of view of the camera 60 be determined suchthat the target object is included within the angle of view in a casewhere the distance to the object during normal use corresponds to theset standards of the maximum and minimum distances.

In general, human beings generally have a visual field angle ofapproximately 200 degrees in the horizontal direction and approximately125 degrees in the vertical direction. Particularly, an effective visualfield within the visual field angle that is excellent in informationreceptivity is approximately 30 degrees in the horizontal direction andapproximately 20 degrees in the vertical direction. It has been assumedthat a stable gazing visual field where a human being can quickly stablysee a gazing point is approximately 60 to 90 degrees in the horizontaldirection and approximately 45 to 70 degrees in the vertical direction.In the case of the target object in which the gazing point is in frontof the user, the effective visual field in the visual field of the useris approximately 30 degrees in the horizontal direction andapproximately 20 degrees in the vertical direction with respect torespective lines of sights of the right eye RE and the left eye LE ascenter lines. The stable gazing visual field is from approximately 60degrees to 90 degrees in the horizontal direction and from approximately45 degrees to 70 degrees in the vertical direction. The visual fieldangle is approximately 200 degrees in the horizontal direction andapproximately 125 degrees in the vertical direction. The visual field inwhich the user actually visually recognizes an object through the rightlight guiding plate 26 and the left light guiding plate 28 can bereferred to as an actual Field Of View (FOV). In the configuration ofthe exemplary embodiment illustrated in FIG. 1 and FIG. 2, the actualfield of view corresponds to the actual field of view in which the uservisually recognizes an object through the right light guiding plate 26and the left light guiding plate 28. The actual field of view isnarrower than the visual field angle and the stable gazing visual field,but wider than the effective visual field.

The camera 60 captures an image of an area including outside scenerythat is visually recognizable together with an image to be displayed bythe image display unit 20. It is preferred that the angle of view of thecamera 60 be capable of capturing an area wider than the visual field ofthe user. Specifically, it is preferred that the angle of view be widerthan at least the effective visual field of the user. Further, it ispreferred that the angle of view be wider than the actual visual fieldof the user. It is further preferred that the angle of view be widerthan the stable gazing visual field of the user, and it is mostpreferred that the angle of view be wider than the visual field of theboth eyes of the user.

The camera 60 may include a so-called wide-angle lens as an imaging lensto enable imaging of a wide angle of view. The wide angle lens mayinclude a lens called a super wide angle lens or a semi-wide angle lens,or may be a single focal lens or a zoom lens. The camera 60 may includea lens group formed of a plurality of lenses.

FIG. 4 is a block diagram illustrating a configuration of each partforming the HMD 100.

The control device 10 includes the main processor 140 executing programsto control the HMD 100. The memory 121 and a non-volatile storage unit123 are coupled to the main processor 140. As sensors, a six-axis sensor111 and a magnetic sensor 113 are coupled to the main processor 140. AGPS receiver 115, a communication unit 117, a sound codec 182, anexternal connector 184, an external memory interface 186, a USBconnector 188, a sensor hub 192, and an FPGA 194 are coupled to the mainprocessor 140. The components function as an interface with externaldevices. The LED indicator 12, an LED display unit 17, a vibrator 19,the operation unit 110, and the power supply unit 130 are coupled to themain processor 140.

The main processor 140 is mounted on a controller substrate 120 builtinto the control device 10. On the controller substrate 120, in additionto the main processor 140, the memory 121, the non-volatile storage unit123, the six-axis sensor 111, the magnetic sensor 113, the GPS receiver115, the communication unit 117, the sound codec 182, and the like arefurther mounted. In the exemplary embodiment, the external connector184, the external memory interface 186, the USB connector 188, thesensor hub 192, the FPGA 194, and an interface 196 are mounted on thecontroller substrate 120.

In a case where the main processor 140 executes a control program, thememory 121 forms a work area temporarily storing the control program tobe executed and data to be processed. The non-volatile storage 123includes a flash memory and an embedded multi media card (eMMC). Thenon-volatile storage 123 stores programs to be executed by the mainprocessor 140 and various data to be processed by the main processor 140executing the programs.

In FIG. 4, a configuration in which the function of the control device10 is achieved by one main processor 140, but the function of thecontrol device 10 may be achieved by a plurality of processors orsemiconductor chips. For example, the controller substrate 120 mayfurther be equipped with a co-processor such as a System-on-a-Chip(SoC), a Micro Control Unit (MCU), and a Field-Programmable Gate Array(FPGA). The control device 10 may perform various controls by causingboth the main processor 140 and the co-processor to cooperate with eachother or selectively using one of both.

The six-axis sensor 111 is a motion sensor (inertial sensor) including athree-axis acceleration sensor and a three-axis gyro (angular velocity)sensor. An Inertial Measurement Unit (IMU) including the above-mentionedsensors as a module may be adopted as the six-axis sensor 111. Themagnetic sensor 113 is a three-axis geomagnetic sensor, for example.

The six-axis sensor 111 and the magnetic sensor 113 output detectedvalues to the main processor 140 in accordance with sampling frequenciesdesignated in advance. In response to a request from the main processor140, the six-axis sensor 111 and the magnetic sensor 113 outputs thedetected values to the main processor 140 at the timings designated bythe main processor 140.

The GPS receiver 115 includes a GPS antenna (not illustrated), andreceives a GPS signal transmitted from a GPS satellite. The GPS receiver115 outputs the received GPS signal to the main processor 140. The GPSreceiver 115 measures a signal strength of the received GPS signal, andoutputs the measurement value to the main processor 140. As the signalstrength, for example, information such as a Received Signal StrengthIndication (RSSI), an electric field strength, a magnetic fieldstrength, and a Signal-to-Noise ratio (SNR) may be used.

The communication unit 117 performs wireless communications with anexternal device. The communication unit 117 includes an antenna, a RFcircuit, a baseband circuit, and a communication control circuit, or thelike, or is formed of a device with the above-mentioned componentsintegrated each other. The communication unit 117 performs wirelesscommunications conforming standards such as Bluetooth and a wireless LAN(including Wi-Fi), for example.

The sound interface 180 is an interface receiving and outputting soundsignals. In the exemplary embodiment, the sound interface 180 includesthe connector 46 (FIG. 1) provided with the coupling cable 40. Theconnector 46 is coupled to the headset 30. The sound signal output fromthe sound interface 180 is received by the right earphone 32 and theleft earphone 34. Then, the right earphone 32 and the left earphone 34outputs the sound. The microphone 63 provided with the headset 30collects sound, and outputs a sound signal to the sound interface 180.The sound signal input from the microphone 63 to the sound interface 180is received by the external connector 184.

The sound codec 182 is coupled to the sound interface 180, and encodesand decodes sound signals input and output via the sound interface 180.The sound codec 182 may include an A/D converter for converting ananalog sound signal into digital sound data and a D/A converter forperforming the opposite conversion. For example, the HMD 100 accordingto the exemplary embodiment outputs sounds from the right ear piece 32and the left ear piece 34, and collects sounds from the microphone 63.The sound codec 182 converts digital sound data output from the mainprocessor 140 into an analog sound signal, and outputs the analog soundsignal via the sound interface 180. The sound codec 182 converts ananalog sound signal input to the sound interface 180 into digital sounddata, and outputs the digital sound data to the main processor 140.

The external connector 184 is a connector for coupling the mainprocessor 140 to an external device that performs communications withthe main processor 140. For example, the external connector 184 couplesthe external device to the main processor 140, and is an interface forestablishing coupling to the external device in a case where a programexecuted by the main processor 140 is debugged or a log of an operationof the HMD 100 is collected.

The external memory interface 186 is an interface that is capable ofbeing coupled to a portable memory device, and includes, for example, amemory card slot mounted with a card-type recording medium for readingdata and an interface circuit. In this case, a size, a shape, and astandard of the card-type recording medium is not limited, and may bechanged as appropriate.

The universal serial bus (USB) connector 188 includes a connectorconforming to the USB standard, and an interface circuit. The USBconnector 188 is capable of being coupled to a USB memory device, asmart phone, and a personal computer. the size and shape of the USBconnector 188, as well as the version of the USB standard that ismatched, may be selected as appropriate.

The sensor hub 192 and the FPGA 194 are coupled to the image displayunit 20 via the interface (I/F) 196. The sensor hub 192 acquiresdetected values of the various sensors included in the image displayunit 20, and outputs the detected values to the main processor 140. TheFPGA 194 processes data to be transmitted and received between the mainprocessor 140 and the components of the image display unit 20, and alsoexecutes transmissions via the interface 196.

The LED indicator 12 turns on and off in response to the operation stateof the HMD 100. The LED display unit 17 controls turning-on andturning-off of the LED indicator 12 in accordance with the control ofthe main processor 140. The LED display unit 17 may include LEDs (notillustrated) arranged directly under the track pad 14 and a drivecircuit for turning on the LEDs. In this case, the LED display unit 17turns on, blinks and turns off the LEDs in accordance with the controlof the main processor 140.

The vibrator 19 includes a motor (not illustrated) and an eccentricrotor (not illustrated), and may include other necessary components. Thevibrator 19 generates vibration by causing the above-mentioned motor torotate in accordance with the control of the main processor 140. Forexample, in a case where an operation on the operation unit 110 isdetected, a case where the power supply of the HMD 100 is turned on oroff, or any other cases, the HMD 100 causes the vibrator 19 to generatevibration in a predetermined vibration pattern.

The operation unit 110 includes the operation elements 13 and the trackpad 14. The operation elements 13 includes the operation buttons 11, theup and down keys 15, the changeover switch 16, and the power switch 18.In a case where the operation elements 13 and the track pad 14 areoperated, the operation unit 110 outputs operation signals includingidentification information of the operation elements 13 and the trackpad 14 that are operated and information indicating the receivedoperation contents to the control unit 150.

The control device 10 includes the power supply unit 130, and operatesby electric power supplied from the power supply unit 130. The powersupply unit 130 includes a chargeable battery 132 and a power supplycontrol circuit 134 for detecting a remaining amount of the battery 132and controlling charge of the battery 132. The power supply controlcircuit 134 is coupled to the main processor 140, and outputs thedetected value of the remaining amount of the battery 132 and thedetected value of a voltage to the main processor 140. Power may besupplied from the control device 10 to the image display unit 20, basedon the power supplied by the power supply unit 130. The main processor140 may be configured to control a state of power supply from the powersupply unit 130 to the components of the control device 10 and the imagedisplay unit 20.

The right display unit 22 and the left display unit 24 of the imagedisplay unit 20 are coupled to the control device 10. As illustrated inFIG. 1, in the HMD 100, the coupling cable 40 is coupled to the leftholding part 23. A wire coupled to the coupling cable 40 is arranged inthe image display unit 20 to couple the right display unit 22 and theleft display unit 24 to the control device 10.

The right display unit 22 includes a display unit substrate 210. On thedisplay unit substrate 210, an interface (I/F) 211 to be coupled to theinterface 196, a receiver (Rx) 213 for receiving data input from thecontrol device 10 via the interface 211, and an EEPROM 215 are mounted.

The interface 211 couples the receiver 213, the EEPROM 215, thetemperature sensor 217, the camera 60, an illuminance sensor 65, and anLED indicator 67 to the control device 10.

The Electrically Erasable Programmable Read-Only Memory (EEPROM) 215stores the various data in a form readable by the main processor 140.The EEPROM 215 stores data about light-emitting characteristics anddisplay characteristics of the OLED units 221 and 241 provided with theimage display unit 20, and data about sensor characteristics of theright display unit 22 or the left display unit 24, for example.Specifically, the EEPROM 215 stores parameters regarding Gammacorrection performed by the OLED units 221 and 241, data used tocompensate for the detected values of the temperature sensors 217 and239, and the like. These data are generated when the HMD 100 isinspected upon shipping from a factory, and written in the EEPROM 215.After shipping, the main processor 140 may perform processing throughuse of the data of the EEPROM 215.

The camera 60 captures an image in accordance with a signal input viathe interface 211, and outputs the captured image data or a signalindicative of the result of imaging to the control device 10.

As illustrated in FIG. 1, the illuminance sensor 65 is provided on theend ER of the front frame 27, and is arranged so as to receive outsidelight coming from the front side of the user wearing the image displayunit 20. The illuminance sensor 65 outputs a detected valuecorresponding to the amount of received light (intensity of receivedlight).

As illustrated in FIG. 1, the LED indicator 67 is arranged in thevicinity of the camera 60 on the end ER of the front frame 27. The LEDindicator 67 is configured to turn on while the camera 60 is capturingimages to notify that the capturing is in progress.

The temperature sensor 217 detects a temperature, and outputs a voltagevalue or a resistance value corresponding to the detected temperature asa detected value. The temperature sensor 217 is mounted on the backsurface of the OLED panel 223 (FIG. 2). The temperature sensor 217 andthe OLED drive circuit 225 may be mounted on a single substrate, forexample. With this configuration, the temperature sensor 217 mainlydetects the temperature of the OLED panel 223.

The receiving unit 213 receives data transmitted from the main processor140 via the interface 211. In a case where the receiver 213 receives theimage data of the image to be displayed by the OLED unit 221, thereceiver 213 outputs the received image data to the OLED drive circuit225 (FIG. 2).

The left display unit 24 includes a display unit substrate 230. On thedisplay unit substrate 230, an interface (I/F) 231 to be coupled to theinterface 196 and a receiver (Rx) 233 for receiving data input from thecontrol device 10 via the interface 231 are mounted. A six-axis sensor235 and a magnetic sensor 237 are mounted on the display unit substrate210. The interface 231 couples the receiver 233, the six-axis sensor235, the magnetic sensor 237, and the temperature sensor 239 to thecontrol device 10.

The six-axis sensor 235 is a motion sensor (inertial sensor) including athree-axis acceleration sensor and a three-axis gyro (angular velocity)sensor. An IMU including the above-mentioned sensors as a module may beadopted as the six-axis sensor 235. The magnetic sensor 237 is athree-axis geomagnetic sensor, for example.

The temperature sensor 239 detects a temperature, and outputs a voltagevalue or a resistance value corresponding to the detected temperature asa detected value. The temperature sensor 239 is mounted on the backsurface of the OLED panel 243 (FIG. 2). The temperature sensor 239 andthe OLED drive circuit 245 may be mounted on a single substrate, forexample. With this configuration, the temperature sensor 239 mainlydetects a temperature of the OLED panel 243. The temperature sensor 239may be built into the OLED panel 243 or the OLED drive circuit 245. Theabove-mentioned substrate may be a semiconductor substrate.Specifically, in a case where the OLED panel 243 is mounted as anSi-OLED together with the OLED drive circuit 245 and the like to form anintegrated circuit on an integrated semiconductor chip, the temperaturesensor 239 may be mounted on the semiconductor chip.

The camera 60, the illuminance sensor 65, and the temperature sensor 217in the right display unit 22, and the six-axis sensor 235, the magneticsensor 237, and the temperature sensor 239 in the left display unit 24are coupled to the sensor hub 192.

The sensor hub 192 sets and initializes sampling periods of the sensorsin accordance with the control of the main processor 140. The sensor hub192 applies a current to each sensor, transmits control data, andacquires detected values in accordance with the sampling period of thesensor. At a preset timing, the sensor hub 192 outputs detected valuesof the sensors included in the right display unit 22 and the leftdisplay unit 24 to the main processor 140. The sensor hub 192 may have afunction of temporarily storing detected values of the sensors takingoutput timing to the main processor 140 into consideration. The sensorhub 192 may have a function of adapting to the differences in a signalform or data form of the output values from the sensors, specifically, afunction of converting the output value of each sensor to data in anintegrated data form, and outputting the converted data to the mainprocessor 140.

The sensor hub 192 starts and stops applying of current to the LEDindicator 67 in accordance with the control of the main processor 140,and turns on or blinks the LED indicator 67 in accordance with a timingat which the camera 60 starts or completes capturing an image.

FIG. 5 is a function block diagram of a storage unit 160 and the controlunit 150 forming a control system of the control device 10. The storageunit 160 illustrated in FIG. 5 is a logical storage unit formed of thenon-volatile storage 123 (FIG. 4) and the EEPROM 215.

The storage unit 160 stores an application program 161, process flowdata 162, and content data 163. The process flow data 162 corresponds to“control data” in the present invention.

The application program 161 is an application program having a specificfunction executed on an OS 151. The application program 161 is a programfor causing the HMD 100 to execute a process flow created with theexternal device. The application program 161 may be an applicationprogram dedicated for executing a process flow, or may be ageneral-purpose program such as a web browser. A specific configurationof the application program 161 can be appropriately selected in view ofa specification of the process flow data 162.

The process flow data 162 may be general-purpose data executable byvarious devices including the HMD 100 (e.g., a device without the camera60) or the HMD 100 with a different type or specification. The processflow data 162 will be described later in detail.

The content data 163 is data of contents including images and movies tobe displayed on the image display unit 20 in accordance with the controlof the control unit 150, and the data includes image data and moviedata. The content data 163 may include music and sound data.

The control unit 150 executes various processing through use of the datastored in the storage unit 160, and controls the HMD 100. The controlunit 150 includes, as function blocks, the Operating System (OS) 151, animage processing unit 152, the imaging control unit 153, an inputdetection unit 154, a communication control unit 155, a display controlunit 156, an information acquisition unit 157, and a process flowexecution unit 158. The function blocks indicate the functions achievedby executing the control program by the main processor 140 as blocks forconvenience. Thus, the function blocks do not refer to specificapplication programs or hardware.

The function of the OS 151 is a function of the control program storedin the storage unit 160, and other components are functions of theapplication programs executed on the OS 151.

For example, the image processing unit 152 reads out the content data163 from the storage unit 160, and separates synchronizing signals suchas a vertical synchronizing signal VSync, a horizontal synchronizingsignal HSync from the read content data 163. In accordance with cyclesof the vertical synchronizing signal VSync and the horizontalsynchronizing signal HSync that are separated, the image processing unit152 generates a clock signal PCLK through use of a Phase Locked Loop(PLL) circuit (not illustrated) and the like. The image processing unit152 may apply, on image data included in the content data 163, variousimage processing such as resolution conversion, luminance adjustment,saturation adjustment, and 2D/3D conversion processing, as needed.

The image processing unit 152 develops the image data after the imageprocessing in a DRAM in the storage unit 160 for each frame being adisplay unit of an image. A region of the DRAM in which one frame of theimage data is developed is referred to as a frame region in thefollowing description. The image processing unit 152 reads out the imagedata from the frame region, and causes the image display unit 20 todisplay the read image data.

The image processing unit 152 may be achieved by the main processor 140executing a program or may be achieved by separate hardware from themain processor 140 (e.g., Digital Signal Processor (DSP)).

The imaging control unit 153 controls the camera 60 to capture an imageto generate captured image data. The imaging control unit 153 causes thestorage unit 160 to store the generated captured image data. In a casewhere the camera 60 is configured as a camera unit including a circuitgenerating the captured image data, the imaging control unit 153acquires the captured image data from the camera 60, and causes thestorage unit 160 to store the captured image data.

The input detection unit 154 detects an input by an operator, based onoperation data input from the operation unit 110.

The communication control unit 155 controls the communication unit 117to execute data communications with another device.

The display control unit 156 controls the right display unit 22 and theleft display unit 24 of the image display unit 20 to control at leastone of a display position and a display state of an object. The objectis an image to be displayed in the display region VR by the HMD 100, andspecifically, the object includes a character, a character string, asymbol, a figure, a number, an icon, and a button. The object includes aframe body surrounding at least one of a character, a character string,a symbol, a figure, a number, an icon, and a button.

The display position of the object controlled by the display controlunit 156 is a display position of the object in the display region VR,and more specifically is a display position in a depth direction of thedisplay region VR. “Depth” refers to a distance from the HMD 100 in adirection in which the user faces under the state in which the userwears the image display unit 20 on the head. Moreover, “depth direction”refers to a direction in which the user faces under the state in whichthe user wears the image display unit 20 on the head, that is, anorientation of the right display unit 22 and the left display unit 24,and a Z-axis direction illustrated in FIG. 1. In the exemplaryembodiment, with the Z axis, the position of the HMD 100 is at theorigin of the Z-axis. As a distance from the HMD 100 increases, acoordinate of the Z axis increases.

In a case where the object is, for example, a character string or anumber, the display state of the object controlled by the displaycontrol unit 156 corresponds to control of a color, a font type, and afont size of the character string or the like. In a case where theobject is a symbol, a figure, an icon, a button or the like, the displaycontrol unit 156 controls a color or a size of the object or the like.

FIG. 6 is a view illustrating a relationship between a size of areference region and a size of an emitting region.

Here, with reference to FIG. 6 and FIG. 7, control of a display positionof an image in the depth direction is described.

The reference region refers to an entire region or a part of the regionin which each of the OLED unit 221 included in the right display unit 22and the OLED unit 241 included in the left display unit 24 can form animage. The reference region may refer to the entire region or a part ofthe region in which the right display unit 22 and the left display unit24 can emit the image light to the half mirrors 261 and 281,respectively. The reference region may refer to a region in which theimage light can be emitted from the half mirrors 261 and 281 to theright eye RE and the left eye LE by using the light emitted from theright display unit 22 and the left display unit 24, respectively. Thereference region may refer to a region in which the half mirrors 261 and281 form a virtual image.

The emitting region refers to a region from which the image light isactually emitted from the right display unit 22 and the left displayunit 24.

In the example of FIG. 6, a reference region MA1 in the right displayunit 22 corresponding to the right eye RE of the user has a size of xpixel×y pixel. Similarly, a reference region MA2 in the left displayunit 24 corresponding to the left eye LE of the user has a size of xpixel×y pixel.

In FIG. 6, the emitting region is indicated with hatched lines. In theexemplary embodiment, the emitting region is smaller than the referenceregion in the horizontal direction (transverse direction). Specifically,an emitting region CA1 in the right display unit 22 corresponding to theright eye RE of the user has a size of (x−n) pixel×y pixel, and is setsmaller than the reference region MA1 in the horizontal direction.Similarly, an emitting region CA2 in the left display unit 24corresponding to the left eye LE of the user has a size of (x−n) pixel×ypixel, and is set smaller than the reference region MA2 in thehorizontal direction.

For example, in a case where a distance in the Z-axis direction fordisplaying the object is adjusted to infinity, a moving amount of theemitting region is “0 (pixel)”. As a result, as illustrated in FIG. 6,an image IM1 visually recognized by the user with the right eye RE,based on the image light emitted from the emitting region CA1 and animage IM2 visually recognized by the user with the left eye LE, based onthe image light emitted from the emitting region CA2 are in a state forparallel viewing. Thus, as a distance in the Z-axis direction fordisplaying the object, infinity can be achieved.

FIG. 7 is an explanatory view for illustrating an image formed with theimage light emitted from the right display unit 22 and the left displayunit 24 in a case where a distance from the right eye RE and the lefteye LE is La.

A moving amount of the emitting regions in a case of adjusting thedistance to La is assumed to be “n (pixel).” The display control unit156 moves the emitting region CA1 in the right display unit 22corresponding to the right eye RE of the user and the emitting regionCA2 in the left display unit 24 corresponding to the left eye LEtogether by n pixel in directions toward the user's eyebrow (indirections of the outlined arrows in FIG. 7). As a result, the image IM1visually recognized by the user with the right eye RE, based on theimage light emitted from the emitting region CA1 and the image IM2visually recognized by the user with the left eye LE, based on the imagelight emitted from the emitting region CA2 form an image at a position(IM) away from the eyes of the user by the distance La. As a result, thedisplay position of the object in the Z-axis direction is adjusted to atarget position.

As described above, an image formed with image light emitted from onedisplay unit (22 or 24) and an image formed with image light emittedfrom the other display unit (22 or 24) moves together to be close to (oraway from) each other. As a result, a convergence angle of the right eyeRE and the left eye LE of the user with respect to the image formed withboth the image light is changed to increase (decrease). Thus, the userusing the HMD 100 can sense the image close to (away from) the user.Therefore, the display position in the Z-axis direction can be adjustedby adjusting the convergence angle of the right eye RE and the left eyeLE of the user.

The information acquisition unit 157 reads out the data from theexternal device coupled via the external connector 184, the externalmemory I/F unit 186, and the USB connector 188. In a case where, forexample, it is detected that a storage device is coupled to the I/F unit186, the information acquisition unit 157 reads out data from thestorage device, and causes the storage unit 160 to store the data. Withthis, the HMD 100 can read out and use the process flow data 162 storedin the storage device.

The process flow execution unit 158 executes the application program 161stored in the storage unit 160, and executes a process flow inaccordance with the process flow data 162.

Here, the process flow and the process flow data 162 are described.

The process flow is so-called work flow representing a work operationincluding a plurality of tasks. The process flow is a specific exampleof an operation flow. The operation flow includes a basic operation or aplurality of basic operations, and represents a sequence in which anorder of execution of the basic operations is defined. For example, thebasic operations include outputting of information such as a display,inputting of information, and processes such as a determination. A unitof operation is used as a unit when creating and editing an operationflow. The unit of operation is not particularly limited to this.Moreover, a unit of operation is used as a unit when creating andediting an operation flow. The unit of operation is not particularlylimited to this. Specifically, when a process flow is created andedited, a content of each basic operation is not edited. Excluding thispoint of view, the basic operations are not limited. Accordingly. acontent of each basic operation may be determined for convenience ofediting an operation flow. A basic operation may contain a plurality ofoperations or processes, for example.

The process flow includes a plurality of tasks performed by an operatorto achieve a set objective, and includes a process for outputtinginformation to the operator during the plurality of tasks is performed.One unit of work performed by an operator and one output of informationto the operator each corresponds to an operation step in an operationflow.

The process flow data 162 are data indicating a work process of work tobe executed by the operator. The process flow data 162 includes workblocks included in the process flow, an order of execution of the workblocks, and setting contents set for each of the work blocks, and mayinclude other information. The work blocks correspond to “operationsteps” in the present invention.

The setting contents set for each of the work blocks include a processto be executed by the control unit 150. The process to be executed bythe control unit 150 includes a process of displaying a screen includingan image and a text as the information to the operator, and a process ofreading a 2D code such as a QR code (registered trademark), receivingtexts, and receiving input of a work result indicating “OK” or “error”.The work blocks include inputting of a selection using a check box andinputting of a selection using a radio button.

The work blocks include, for example, a procedure display block, areading block, an inputting block, a determination block, a selectiveinput block, and a completion block.

The procedure display block is a block for causing the image displayunit 20 to display an image or a text indicating a work procedure in thedisplay region. The operator performs the work with reference to theimage or the text displayed by the image display unit 20 in the displayregion.

The reading block is a block for causing the camera 60 to capture animage and reading a 2D code such as a QR code (registered trade mark)and a bar code from the captured image data of the camera 60.

For example, a reading frame for a 2D code and a guiding display forguiding the operator to set the 2D code within the reading frame arecaused to be displayed by the control unit 150. In a case where an imageof a 2D code is detected in the reading frame, the control unit 150cause the storage unit 160 to store the detected image of the 2D code,and analyzes code information included in the 2D code.

The inputting block is a block for receiving an input from the operatorby the operation unit 110 or the like. A method of receiving an inputfrom the operator is not limited to a method using the operation unit110. For example, a text recognition process using the captured imagedata of the camera 60 and a sound recognition process for soundcollected with the microphone 63 may be used to detect various inputincluding text. The read data obtained by reading a bar code may beacquired from a bar-code reader coupled through the external connector184, and the acquired data may be received as the input by the operator.A motion of the operator obtained by detection values of the six-axissensors 235 and 111 may be received as the input by the operator.

The determination block is a block for receiving an execution result ofthe work from the operator. For example, the operator performs work inaccordance with the work procedure displayed in the procedure displayblock. As a work result, a work result indicating “OK” or “error” isinput by the operation unit 110 or the like. In a case where theoperation unit 110 receives input indicating “OK,” the control unit 150makes a positive determination. In this case, the control unit 150executes a work block corresponding to the positive determination, forexample, displays a screen including a next work content by executingthe procedure display block. In a case where the operation unit 110receives input indicating “error,” the control unit 150 makes a negativedetermination. In this case, the control unit 150 executes a work blockcorresponding to the negative determination, for example, displays thesame work content again to cause the same work to be executed again, orcaptures an image of a work result by the camera 60.

The selective input block is a block for receiving input of a selectionusing a check box or a radio button. In a case where the selective inputblock is executed, the control unit 150 displays a check box or a radiobutton so that the operator can visually recognized the check box or theradio button, to receive an operation of selecting the check box or theradio button. In response to the reception of the selected operation,the control unit 150 executes a work block corresponding to the receivedselection.

The completion block is a block indicating an end of the process flow.

The setting contents set for each of the work blocks of the process flowdata 162 include setting information in which objects to be displayed inan emphasized manner are set. The setting information in which theobjects to be displayed in an emphasized manner can be set for each ofthe work blocks. Thus, the objects to be displayed in an emphasizedmanner can be changed for each of the work blocks. The settinginformation corresponds to “selection information” in the presentinvention. The display in the emphasized manner refers to a display inwhich a display position of the set object in the Z-axis direction isdifferent from display positions of other objects. The set object isdisplayed at a position different from the position of other objects inthe Z-axis direction. Accordingly, the set object is caused to bedistinguished from other objects, and hence the set object is preventedfrom being missed.

The set information includes at least information specifying the objectsto be displayed in an emphasized manner. That is, the set informationincludes information specifying any of a character, a character string,a symbol, a figured, a number, an icon, and a button. In addition to theabove, the set information may include information for designating acolor and a size at the time of displaying the object.

In a case where the object includes a character or a character string,the set information may include information indicating a font type, afont size, boldness of a character, and a background color of acharacter. The work blocks may have different setting information fromeach other.

FIG. 8 is a view for illustrating a display example of the displayregion VR. Particularly, in FIG. 8, there is illustrated a state inwhich an image or a text indicating the work procedure is displayed inthe display region VR by causing the control unit 150 to execute theprocedure display block included in the process flow data 162.

In a case where the object set in the setting information in advance isdetected in a document or an image to be displayed in the display regionVR, the control unit 150 displays the set object, the display positionof the set object in the Z-axis direction being different from displaypositions of other objects.

In the following description, as a specific example of the object set inadvance, a case where a character string is set is described.Hereinafter, the character string set in advance is referred to as anemphasized character string 300.

For example, description is given of a case where characters saying“Please confirm that three ⋅× bolts are fastened” are displayed in thedisplay region VR. Description is given when a character string “three”is assumed to be the emphasized character string 300. In this case, thedisplay control unit 156 displays the character “three” at a position inthe Z-axis direction different from positions of other characterstrings. Specifically, the display control unit 156 adjusts the displayposition of the character “three” being the emphasized character string300 in the Z-axis direction so that the character string “three” isdisplayed nearer than the display positions of other character strings.

FIG. 9 is a view illustrating the emphasized character string 300displayed in the display region VR.

A rectangular frame body 331 indicated with a broken line in FIG. 9 hasa plane parallel with an X axis and a Y axis (hereinafter, referred toas an XY plane). The display position of the frame body 331 in theZ-axis direction is the same position as the display position of theemphasized character string 300. That is, the frame body 331 correspondsto a surface on which the emphasized character string 300 is displayed.The X axis corresponds to the horizontal direction in the display regionVR, and the Y axis corresponds to the vertical direction in the displayregion VR.

A rectangular frame body 332 indicated with a solid line in FIG. 9 hasthe XY plane, and the display position of the frame body 332 in theZ-axis direction is the same position as the display positions ofcharacter strings other than the emphasized character string 300. Thatis, the frame body 332 is a plane on which character strings other thanthe emphasized character string 300 are displayed. Hereinafter, thecharacter strings other than the emphasized character string 300 arereferred to as “other character strings.”

The display control unit 156 displays the characters other than theemphasized character string 300 at positions within a range of adistance from 1 m to 3 m in the Z-axis direction. That is, the framebody 332 is at a position within the range of the distance from 1 m to 3m in the Z-axis direction. The display control unit 156 displays theemphasized character string 300 at a position in front of or behindother character strings by 10 cm. In FIG. 9, there is illustrated anexample in which the frame body 331 is displayed at a position nearerthan the position of the frame body 332 by 10 cm.

FIG. 10 is a view illustrating a display position of the characterstring in the Z-axis direction. Particularly, FIG. 10 illustrates astate in which only the emphasized character string 300 is displayed.

The display control unit 156 causes the position of the emphasizedcharacter string 300 in the Z-axis direction to be changed every certaintime. The certain time corresponds to “unit time” in the presentinvention. The certain time can be set and changed arbitrarily inaccordance with the user's preference.

In FIG. 10, the emphasized character string 300 indicated with thebroken lines is displayed at a coordinate “Z1” in the Z-axis direction.Although not illustrated in FIG. 10, the coordinate “Z1” is a positionat which other character strings are displayed.

The emphasized character string 300 indicated with the solid lines isdisplayed at a coordinate “Z2” in the Z-axis direction. The coordinate“Z1” has a value larger than a value of the coordinate “Z2.” That is, ina case where the emphasized character string 300 is displayed at thecoordinate “Z1,” the emphasized character string 300 is displayed at aposition (far side) away from the HMD 100 as compared to the emphasizedcharacter string 300 displayed at the coordinate “Z2.”

The display control unit 156 displays the emphasized character string300 alternately at the display position of the coordinate “Z1” and thedisplay position of the coordinate “Z2” every certain time. The displaycontrol unit 156 changes the display state of the emphasized characterstring 300 in this manner, to thereby emphasize the display of theemphasized character string 300.

FIG. 11 is a view illustrating a display position of the characterstring in the Z-axis direction. Particularly, in FIG. 11, there isillustrated a state in which the character string formed of twocharacters, a number and a Japanese Kanji character (Chinese character),meaning “three” being the emphasized character string 300 and anothercharacter string formed of three Japanese Katakana characters meaning“bolts” are displayed.

In FIG. 11, the other character string with three Katakana charactersmeaning “bolts” illustrated with the solid lines is displayed at acoordinate “Z4” in the Z-axis direction. In FIG. 11, the emphasizedcharacter string 300 illustrated with the broken lines is displayed at acoordinate “Z3” in the Z-axis direction, and the emphasized characterstring 300 illustrated with the solid lines are displayed at acoordinate “Z5” in the Z-axis direction. The coordinate “Z3” has a valuelarger than that of the coordinate “Z4,” and the coordinate “Z4” has avalue larger than that of the coordinate “Z5.” That is, the emphasizedcharacter string 300 displayed at the coordinate “Z3” is displayed at aposition (far side) away from the HMD 100 as compared to the othercharacter string displayed at the coordinate “Z4.” The emphasizedcharacter string 300 displayed at the coordinate “Z5” is displayed at aposition (user side) close to the HMD 100 as compared to the othercharacter string displayed at the coordinate “Z4.”

The display control unit 156 displays the emphasized character string300 alternately at the display position of the coordinate “Z3” and thedisplay position of the coordinate “Z5” every certain time.

In FIG. 12A, the other character string “bolts” indicated with the solidlines is displayed at a coordinate “Z7” in the Z-axis direction. In FIG.12A, the character string formed of the number and the Japanese Kanjicharacter meaning “three” illustrated with the solid lines is a firstemphasized character string 301, and is displayed at a coordinate “Z8.”In FIG. 12A, a character string formed of two Kanji characters meaning“confirm” illustrated with the solid lines is a second emphasizedcharacter string 302, and is displayed at a coordinate “Z6.” The firstemphasized character string 301 corresponds to “first object” in thepresent invention, and the second emphasized character string 302corresponds to “second object” in the present invention. The firstemphasized character string 301, the second emphasized character string302, and other character strings are displayed at positions notoverlapping with each other in the Z-axis direction. The firstemphasized character string 301, the second emphasized character string302, and the other character strings are displayed at the positions notoverlapping with each other in the Z-axis direction, and thus,degradation of visibility of information to be visually recognized bythe user can be prevented.

In FIGS. 12A and 12B, the coordinate “Z6” has a value larger than thatof the coordinate “Z7,” and the coordinate “Z7” has a value larger thanthat of the coordinate “Z8.” In FIG. 12A, the second emphasizedcharacter string 302 with two Kanji characters meaning “confirm”displayed at the coordinate “Z6” is displayed at a position (far side)away from the HMD 100 as compared to the other character string withthree Katakana characters meaning “bolts” displayed at the coordinate“Z7.” In FIG. 12A, the first emphasized character string 301 with anumber and a Kanji character meaning “three” displayed at the coordinate“Z8” is displayed at a position (user side) close to the HMD 100 ascompared to the other character string with three katakana charactersmeaning “bolts” displayed at the coordinate “Z7.”

In FIG. 12B, the character string with a number and a Kanji charactermeaning “three” being the first emphasized character string 301 isdisplayed at the coordinate “Z6” in the Z-axis direction, and thecharacter string with two Kanji characters meaning “confirm” being thesecond emphasized character string 302 is displayed at the coordinate“Z8” in the Z-axis direction. In FIG. 12B, the other character stringwith three Katakana characters meaning “bolts” is displayed at the samecoordinate “Z7” as illustrated in FIG. 12A.

In FIG. 12B, the first emphasized character string 301 with a number anda Kanji character meaning “three” displayed at the coordinate “Z6” isdisplayed at a position (far side) away from the HMD 100 as compared tothe other character string with three Katakana characters meaning“bolts” displayed at the coordinate “Z7.” In FIG. 12B, the secondemphasized character string 302 with two Kanji characters meaning“confirm” displayed at the coordinate “Z8” is displayed at a position(user side) close to the HMD 100 as compared to the other characterstring with three Katakana characters meaning “bolts” displayed at thecoordinate “Z7.”

The display control unit 156 does not change the display position of theother character string with three Katakana characters meaning “bolts” inthe Z-axis direction, and alternately changes the display positions ofthe first emphasized character string 301 and the second emphasizedcharacter string 302 every certain time. That is, the display controlunit 156 controls the right display unit 22 and the left display unit 24so that the state illustrated in FIG. 12A and the state illustrated inFIG. 12B are displayed alternately.

The display control unit 156 changes the display positions of the firstemphasized character string 301 and the second emphasized characterstring 302 in this manner, to thereby emphasize the display of the firstemphasized character string 301 and the second emphasized characterstring 302.

In FIG. 12A and FIG. 12B, the position at the coordinate “Z6” and theposition at the coordinate “Z7” may be the same.

For example, in a case where the first emphasized character string 301is displayed at a position nearer than the second emphasized characterstring 302, the second emphasized character string 302 may be displayedat the same display position as the display position of the othercharacter string with three Katakana characters meaning “bolts.”Similarly, in a case where the second emphasized character string 302 isdisplayed at a position nearer than the first emphasized characterstring 301, the first emphasized character string 301 may be displayedat the same display position as the display position of the othercharacter string with three Katakana characters meaning “bolts.”

FIGS. 13A to 13C are views each illustrating the emphasized characterstring 300 displayed in the display region VR.

The display control unit 156 sets an imaginary axis, and causes theemphasized character string 300 to turn about the set imaginary axis.Specifically, the display control unit 156 switches the display positionof the emphasized character string 300 for each certain time.Accordingly, the display control unit 156 causes the emphasizedcharacter string 300 to be displayed in the display region VR so thatthe emphasized character string 300 turns about the imaginary axis. Theimaginary axis may be set in an arbitrary direction, and the directionmay be the vertical direction, the horizontal direction, and an obliquedirection as long as the user can visually recognize that the displayposition of the emphasized character string 300 is changed.

In FIGS. 13A to 13C, the following case is illustrated. That is, theimaginary axis is set in a direction parallel with the Z-axis directionbeing the direction of the line of sight of the user, and the displayposition of the emphasized character string 300 is changed by causingthe emphasized character string 300 to turn about the set imaginaryaxis.

In FIG. 13A, the emphasized character string 300 before being changed indisplay position is illustrated. In FIG. 13B, there is illustrated astate in which the emphasized character string 300 is caused to turn ina counterclockwise direction (left turning) on the sheet of FIG. 13B. InFIG. 13C, there is illustrated a state in which the emphasized characterstring 300 is caused to turn in a clockwise direction (right turning) onthe sheet in FIG. 13C.

The display control unit 156 changes the display of the emphasizedcharacter string 300 from the state in FIG. 13A to the state in FIG.13B, and further returns the display from the state in FIG. 13B to thestate in FIG. 13A. The display control unit 156 changes the display ofthe emphasized character string 300 from the state in FIG. 13A to thestate in FIG. 13C, and further returns the display from the state inFIG. 13C to the state in FIG. 13A. The display control unit 156 changesthe display position of the emphasized character string 300 as describedabove, and thus, the display of the emphasized character string 300 isemphasized.

FIGS. 14A to 14C are views each illustrating the emphasized characterstring 300 displayed in the display region VR.

In FIGS. 14A to 14C, the following case is illustrated. That is, theimaginary axis is set in a direction parallel with the Y-axis directionbeing the vertical direction, and the emphasized character string 300 iscaused to turn about the set imaginary axis.

In FIG. 14A, the emphasized character string 300 before being changed indisplay position is illustrated. In FIG. 14B, there is illustrated astate in which the emphasized character string 300 is caused to turnabout the imaginary axis in a clockwise direction (right turning) on thesheet of FIG. 14B. In FIG. 14C, there is illustrated a state in whichthe emphasized character string 300 is caused to turn about theimaginary axis in a counterclockwise direction (left turning) on thesheet of FIG. 14C.

Regarding the state of the emphasized character string 300 illustratedin FIG. 14B, of the two characters meaning three being the emphasizedcharacter string 300, the position of the right Kanji character on thesheet of FIG. 14B in the Z-axis direction is displayed at a nearerposition (user side) as compared to the left number “3” on the sheet inthe Z-axis direction.

Regarding the state of the emphasized character string 300 illustratedin FIG. 14C, of the two characters meaning three being the emphasizedcharacter string 300, the position of the left number “3” on the sheetof FIG. 14C in the Z-axis direction is displayed at a nearer position(user side) as compared to the right Kanji character on the sheet in theZ-axis direction.

The display control unit 156 changes the display of the emphasizedcharacter string 300 from the state in FIG. 13A to the state in FIG.13B, and further returns the display from the state in FIG. 13B to thestate in FIG. 13A. The display control unit 156 changes the display ofthe emphasized character string 300 from the state in FIG. 13A to thestate in FIG. 13C, and further returns the display from the state inFIG. 13C to the state in FIG. 13A. The display control unit 156 changesthe display position of the emphasized character string 300 as describedabove, and thus, the display of the emphasized character string 300 isemphasized.

FIGS. 15A to 15C are views each illustrating the emphasized characterstring 300 displayed in the display region VR. FIGS. 15A to 15C aredisplay examples in which a size (length) of one end and a size (length)of another end are changed.

In FIG. 15A, the emphasized character string 300 before being changed insize is illustrated. In FIG. 15B, there is illustrated a state in whicha character size of the emphasized character string 300 is continuouslyreduced as approaching from the right end to the left side on the sheetof FIG. 15B.

In FIG. 15C, there is illustrated a state in which a size of theemphasized character string 300 is continuously reduced as approachingfrom the left end to the right side on the sheet of FIG. 15C.

The display control unit 156 changes the character size of theemphasized character string 300, to change the state from the state inFIG. 15A to the state in FIG. 15B. The display control unit 156continuously returns the character size of the emphasized characterstring 300 which is changed to the state in FIG. 15B, to the state inFIG. 15A.

The display control unit 156 changes the character size of theemphasized character string 300 to change the state from the state inFIG. 15A to the state in FIG. 15C. The display control unit 156continuously returns the character size of the emphasized characterstring 300 which is changed to the state in FIG. 15C, to the state inFIG. 15A. The display control unit 156 changes the display sizes of theright and left sides in the character string as described above.Accordingly, the display of the emphasized character string 300 isemphasized.

FIGS. 16A and 16B are views each illustrating the emphasized characterstring 300 displayed in the display region VR. Particularly, in FIGS.16A and 16B, there is illustrated a case where two-character strings of“three” being the first emphasized character string 301 and “confirm”being the second emphasized character string 302 are selected as theemphasized character string 300.

In FIG. 16A, there is illustrated a state in which the display positionof “three” being the first emphasized character string 301 in the Z-axisdirection is at a nearer position (user side) than the other characterstring. In FIG. 16A, the display position of “confirm” being the secondemphasized character string 302 in the Z-axis direction is the same asthe display position of the other character string.

In FIG. 16B, there is illustrated a state in which the display positionof “confirm” being the second emphasized character string 302 in theZ-axis direction is at a nearer position (user side) than the othercharacter string. In FIG. 16B, the display position of “three” being thefirst emphasized character string 301 in the Z-axis direction is at thesame display position of the other character string.

The display control unit 156 controls the right display unit 22 and theleft display unit 24 so that the first emphasized character string 301and the second emphasized character string 302 are alternately displayedat a nearer position (user side) than the other character string in theZ-axis direction for certain time.

The display control unit 156 changes the display positions of the firstemphasized character string 301 and the second emphasized characterstring 302 in this manner, to thereby emphasize the display of the firstemphasized character string 301 and the second emphasized characterstring 302.

FIG. 17 is a view for illustrating display of the display region VR.Particularly, in FIG. 17, there is illustrated a state in which a framebody 305 surrounds the emphasized character string 300.

In a case where the emphasized character string 300 is detected, thedisplay control unit 156 surround the detected emphasized characterstring 300 with the frame body 305. The display control unit 156 changesthe display position of the frame body 305 surrounding the emphasizedcharacter string 300 in the Z-axis direction every certain time. Thedisplay control unit 156 changes the display of the frame body 305surrounding the emphasized character string 300 in this manner, tothereby emphasize the display of the emphasized character string 300.Note that, in FIG. 17, the frame body 305 surrounding characters isillustrated as the frame body 305. However, the object surrounded by theframe body 305 may include a symbol, a figure, a number, an icon, and abutton. The frame body 305 may surround a plurality of kinds of objectssuch as a figure and a number.

FIG. 18 is a flowchart illustrating an operation of the control unit150.

The control unit 150 reads the process flow data 162 from the storageunit 160, and executes the work blocks included in the process flow data162. Specifically, the control unit 150 selects a work block to beexecuted in accordance with an order of execution included in theprocess flow data 162 (Step S1). Next, the control unit 150 executesprocessing in accordance with the setting content set in the selectedwork block (Step S2). At this time, the control unit 150 analyzes adocument or an image to be displayed as information for the operator,and performs an operation of detecting the emphasized character string300 set in the setting information. The control unit 150 determineswhether the emphasized character string 300 is included in the documentor the image to be displayed (Step S3).

In a case where it is determined that the emphasized character string300 is included (Step S3/YES), the control unit 150 causes the imagedisplay unit 20 to display the document or the image to be displayed inthe display region VR. Further, the control unit 150 changes the displayposition of the detected emphasized character string 300 in the Z-axisdirection (Step S4). That is, the control unit 150 controls the displayposition of the emphasized character string 300 so that the displayposition of the detected emphasized character string 300 in the Z-axisdirection is different from the display position of the other characterstring. After that, the control unit 150 determines whether theprocessing for entire selected work block is completed (Step S5). Thedetermination on whether the processing of the work block is completedmay be made by, for example, an operation received by the operation unit110, or it may be determined that the process is completed in a casewhere the control unit 150 determines that all the process set in thework block is executed. In a case where it is determined that theprocessing for entire work block is not completed, the control unit 150returns to Step S3 (Step S5/NO). The control unit 150 causes the imagedisplay unit 20 to display a document or an image to be newly displayedin the display region VR. In a case where the emphasized characterstring 300 is included in the displayed document or image, the controlunit 150 changes the display position of the detected emphasizedcharacter string 300 in the Z-axis direction. In a case where it isdetermined that the processing for entire work block is completed (StepS5/YES), the control unit 150 terminates the process flow.

A method for displaying the emphasized character string 300 with moreemphasis than the other character string is not limited to theabove-mentioned method. For example, in a case where the emphasizedcharacter string 300 is displayed at a position closer to the HMD 100than the other character string in the Z-axis direction, boldness of theemphasized character string 300 may be changed to be thicker than thatof the other character string. In a case where the emphasized characterstring 300 is positioned at a position away from the HMD 100 than theother character string in the Z-axis direction, the boldness of theemphasized character string 300 may be changed to be thinner than thatof the other character string.

The emphasized character string 300 may be displayed in a colordifferent from a color of the other character string. The emphasizedcharacter string 300 may be displayed in different colors depending on acase where the emphasized character string 300 is displayed at aposition closer to the HMD 100 than the other character string in theZ-axis direction and a case where the emphasized character string 300 isdisplayed at a position away from the HMD 100 than the other characterstring in the Z-axis direction.

The display of the emphasized character string 300 may be controlled sothat at least one of a font type and a font size of the emphasizedcharacter string 300 and at least corresponding one of a font type and afont size of the other character string are different from each other.

FIG. 19 is a view illustrating a display example of the display regionVR.

An image to be displayed in the display region VR by the display controlunit 156 includes an icon 320 and a background image 310. In a casewhere the emphasized character string 300 is displayed in a colordifferent from a color of the other character string, the displaycontrol unit 156 may display the icon 320 in the same color as that ofthe emphasized character string 300. The display control unit 156 maydisplays the background image 310 only for the emphasized characterstring 300. In a case where the background image 310 is displayed forthe emphasized character string 300 and the other character string, thebackground image 310 for the emphasized character string 300 and thebackground image for the other character string may be displayed indifferent colors or patterns. The display control unit 156 may set thecolors of the emphasized character string 300 and the background image310 so that the color of the background image 310 and the color of theemphasized character string 300 are complementary.

With the above-mentioned display, the operator recognizes the emphasizedcharacter string 300 more easily.

The display of the icon 320 may be blinked to indicate that theemphasized character string 300 is included in the document or the imagedisplayed in the display region VR.

The display in which the display position of the emphasized characterstring 300 is changed in the Z-axis direction and the display in whichthe display position of the icon 320 in the Z-axis direction may beperformed at different timings. For example, while the display positionof the emphasized character string 300 in the Z-axis direction is beingchanged, the display position of the icon 320 in the Z-axis direction isnot changed. That is, the display position of the icon 320 is notchanged. In a case where the display position of the icon 320 in theZ-axis direction is changed, the display position of the emphasizedcharacter string 300 in the Z-axis direction is not changed. Thus, thedisplay position of one of the emphasized character string 300 and theicon 320 in the Z-axis direction is changed. Accordingly, it is possibleto enhance an effect of allowing the inclusion of the emphasizedcharacter string 300 in the document or the image to be recognized.

In a case where the display position of the emphasized character string300 in the Z-axis direction is changed, the display control unit 156 mayalso change the display position in the X axis direction and the Y axisdirection.

For example, in a case where the emphasized character string 300 isdisplayed on a user side than the other character string, the displaycontrol unit 156 controls the display position of the emphasizedcharacter string 300 so that the emphasized character string 300 isdisplayed near the center of the display region VR. In a case where theemphasized character string 300 is displayed on a far side than theother character string, the display control unit 156 causes theemphasized character string 300 to be displayed at a position closer toa peripheral part in the display region VR as compared to the case wherethe emphasized character string 300 is displayed on a user side. Thatis, in a case where the display position of the emphasized characterstring 300 is changed from a far side to a user side than the othercharacter string, the emphasized character string 300 is displayed so asto move from the peripheral part to the center part in the displayregion VR. In a case where the display position of the emphasizedcharacter string 300 is changed from a user side to a far side than theother character string, the emphasized character string 300 is displayedso as to move from the center part to the peripheral part in the displayregion VR.

As described above, in the exemplary embodiment, the HMD 100 throughwhich outside scenery can be visually recognized includes the imagedisplay unit 20, the storage unit 160, and the control unit 150.

The image display unit 20 is configured to display objects in thedisplay region VR overlapping with the outside scenery.

The storage unit 160 is configured to store the process flow data 162 asselection information to be used for selecting the objects to bedisplayed in the display region VR.

The control unit 150 is configured to select any of the objects to bedisplayed in the display region VR. based on the setting informationincluded in the process flow data 162, and to control display positionsof the objects in the depth direction of the display region VR so that adisplay position of the selected object is different from a displayposition of another object to be displayed in the display region.

Thus, an oversight of the objects can be reduced, and the informationdisplayed by the HMD 100 can effectively be utilized.

The control unit 150 adjusts the display positions of the objects in thedepth direction by changing a convergence angle between the right eyeand the left eye of the user.

Therefore, the display positions of the objects in the depth directioncan easily be adjusted.

The control unit 150 controls the display positions of the objects inthe depth direction of the display region VR and the display states ofthe objects.

Thus, an oversight of the object is further reduced, and the informationdisplayed by the HMD 100 can effectively be utilized.

The control unit 150 changes the display position of the selected objectin the depth direction every unit time.

Thus, an oversight of the object is further reduced, and the informationdisplayed by the HMD 100 can effectively be utilized.

The control unit 150 changes a display state every unit time between astate in which the selected object is displayed on a user side thananother object in the depth direction and a state in which the selectedobject is displayed on a far side than another object in the depthdirection.

Thus, an oversight of the object is further reduced, and the informationdisplayed by the HMD 100 can effectively be utilized.

The control unit 150 changes the display position of the selected objectabout an imaginary axis set in a predetermined direction with thedisplay region VR as a reference.

Thus, an oversight of the object is further reduced, and the informationdisplayed by the HMD 100 can effectively be utilized.

The control unit 150 sets the imaginary axis to a direction parallelwith the vertical direction of the display region VR. The control unit150 alternately switches the following two display states every unittime. In one display state, one end of the selected object in adirection parallel with the horizontal direction of the display regionVR is displayed on a user side in the depth direction than another endof the selected object in the direction parallel with the horizontaldirection of the display region VR. In the other display state, theother end is displayed on a user side in the depth direction than theone end.

Thus, an oversight of the object is further reduced, and the informationdisplayed by the HMD 100 can effectively be utilized.

The control unit 150 alternately switches the following two displaystates every unit time. In one display state, a length of the selectedobject on the one end side is shorter than a length of the selectedobject on the other end side. In another display state, the length ofthe selected object on the other end side is shorter than the length ofthe selected object on the one end side.

Thus, an oversight of the object is further reduced, and the informationdisplayed by the HMD 100 can effectively be utilized.

In a case where a character string is selected as an object, the controlunit 150 causes the selected character string and the other characterstring to be different in at least any of a color, boldness, abackground color, a font type, and a font size.

Therefore, the user can recognize the character string easily, and anoversight of the information needed for the user can further be reducedwith the HMD 100. Thus, the information displayed by the HMD 100 caneffectively be utilized.

The storage unit 160 stores control data for defining an operation flowincluding a plurality of operation steps for which an order of executionis set in advance.

In a case where the operation flow is executed, the control unit 150switches whether or not to execute control of the display position ofthe object in accordance with the setting of the control data regardingthe operation steps.

Therefore, the user can recognize the object more easily, and anoversight can further be reduced. Therefore, in a case of using the HMD100 for work support, work of the user can effectively be supported.

In a case where a plurality of objects are selected, the control unit150 causes the plurality of selected objects to be sequentiallydisplayed in the display region VR.

Therefore, even when the plurality of objects are selected, an oversightof the objects can be reduced, and the information displayed by the HMD100 can effectively be utilized.

In a case where the first emphasized character string 301 and the secondemphasized character string 302 are selected as the plurality ofobjects, the control unit 150 switches a display state between a firstdisplay state and a second display state.

In the first display state, in the depth direction, the first emphasizedcharacter string 301 is displayed on a user side than another object,and the second emphasized character string 302 is displayed on a farside than another object. In the second display state, in the depthdirection, the second emphasized character string 302 is displayed on auser side than the other object, and the first emphasized characterstring 301 is displayed on a far side than the other object.

Therefore, even when the plurality of objects are selected, an oversightof the objects can be reduced, and the information displayed by the HMD100 can effectively be utilized.

The control unit 150 controls the display position of the selectedobject in the display region VR so that the selected object does notoverlap with another object.

Thus, degradation of visibility of the plurality of objects can beprevented.

The control unit 150 selects an object including any of a character, theicon 320, and the frame body 305 surrounding at least any of thecharacter and the icon 320.

Therefore, in a case where any of the character, the icon 320, and theframe body 305 is displayed by the HMD 100, an oversight of the objectscan be reduced, and the information displayed by the HMD 100 caneffectively be utilized.

The above-mentioned exemplary embodiment is a preferred exemplaryembodiment of the present invention. However, the exemplary embodimentis not limited thereto, and can be modified in various manners withoutdeparting from the scope of the present invention.

For example, in the above-mentioned exemplary embodiment, thetransmissive HMD 100 is exemplified for description. However, the HMD100 may be a non-transmissive head-mounted type display device throughwhich outside scenery cannot be visually recognized. The presentinvention is applicable to a device as long as the device can change adisplay position in a depth direction of a display region. For example,examples of the device to which the present invention is applicableinclude a personal computer (PC), a tablet PC, a smart phone, aprojector, and the like.

The HMD 100 may include an interface (not illustrated) for couplingvarious external devices being supply sources of contents. For example,the interface may be an interface for a wired connection, such as a USBinterface, a micro USB interface, and an interface for a memory card, ormay be a wireless communication interface. The external device in thiscase is an image supply device supplying an image to the HMD 100, andmay be a personal computer (PC), a portable phone terminal, a portablegame machine, or the like. In this case, the HMD 100 can output an imageor sound based on the content data 163 input from such external device.

In the above-mentioned exemplary embodiment, a configuration in whichthe control device 10 is coupled to the image display unit 20 via a wireis exemplified, but the present invention is not limited thereto. Theimage display unit 20 may be coupled to the control device 10wirelessly. As a wireless communication method in this case, the methodexemplified as the communication method for the communication unit 117may be employed, and other communication methods may be employed.

Part of the function of the control device 10 may be provided in theimage display unit 20, and the control device 10 may be achieved by aplurality of devices. That is, the control device 10 is not limited tothe configuration including the case 10A having a box shape. Forexample, in place of the control device 10, the body of the user,clothes of the user, or a device that is wearable by the user may beused. The device that is wearable by the user may be, for example, awatch-type device, a ring-type device, a laser pointer, a mouse, an airmouse, a game controller, a pen-type device, and the like.

Further, in the above-mentioned exemplary embodiment, a configuration inwhich the image display unit 20 and the control device 10 are separatedand coupled via the coupling cable 40 is exemplified. The presentinvention is not limited thereto. The control device 10 and the imagedisplay unit 20 may be integrated to be mounted on the user's head.

As the control device 10, a notebook-type computer, a tablet-typecomputer, or a desktop-type computer may be used. As the control device10, a portable electronic device including a game machine, a portablephone terminal, a smart phone, and a portable media player, and otherdedicated devices may be used.

For example, in place of the image display unit 20, an image displayunit using another method such as an image display unit worn like a hatmay be employed. Such an image display unit is only required to includea display unit corresponding to the left eye LE of the user anddisplaying an image, and a display unit corresponding to the right eyeRE of the user and displaying an image. In place of the image displayunit 20, a head-up display may be used. According to this configuration,the display device can be mounted on a vehicle such as an automobile andan airplane. For example, in a case where the head-up display is mountedin a vehicle, an operation surface corresponding to the operationsurface of the track pad 14 is provided to a steering wheel or the likeof a vehicle.

For example, the head-mounted type display device may be embedded in abody protector such as a helmet. In this case, it may be assumed thatpositions corresponding to a body of a user are positioning portions,and portions that are positioned for the portions are mounted portions.

As the optical system for guiding the image light to the eyes of theuser, a configuration in which a virtual image is formed in a part ofthe right light guiding plate 26 and the left light guiding plate 28 bythe half mirrors 261 and 281 is exemplified. The present invention isnot limited thereto, and an image maybe displayed in the display regionwith area of the entire part or the most part of the right light guidingplate 26 and the left light guiding plate 28. In this case, in theoperation for changing the display position of the image, process forreducing an image size may be included.

The optical elements of the present invention are not limited to theright light guiding plate 26 and the left light guiding plate 28including the half mirrors 261 and 281, and are only required to beoptical components causing the image light to enter the eyes of theuser. Specifically, a diffraction grating, a prism, and a holographicdisplay unit may be used.

At least part of the function blocks illustrated in FIG. 4 and FIG. 5may be achieved by hardware, may be a configuration achieved bycooperation of hardware and software, and is not limited to theconfiguration in which the independent hardware resources are arrangedas illustrated in the drawings. The program executed by the control unit150 may be stored in other storage devices (not illustrated) in thenon-volatile storage unit 123 or the control device 10. A program storedin an external device may be acquired via the communication unit 117 orthe external connector 184 and executed. As a configuration formed inthe control device 10, the operation unit 110 may be formed as a userinterface (UI).

A unit of processing in the flowchart illustrated in FIG. 18 is obtainedby dividing the processing of the control unit 150 of the HMD 100according to the main processing contents for easy understanding. Thus,the present invention is not limited by a method of division into theunit of processing or a name of the unit of processing. The processingof the control unit 150 may be divided into a larger number of units ofprocessing depending on the processing contents, or may be divided sothat one unit of processing includes more processing contents. An orderof processing in the above-mentioned flowchart is not limited to theillustrated example.

The entire disclosure of Japanese Patent Application No. 2017-239265,filed Dec. 14, 2017 is expressly incorporated by reference herein.

What is claimed is:
 1. A head-mounted type display device configured tobe mounted on a head of a user and through which outside scenery isvisible, the head-mounted type display device comprising: a displayconfigured to display objects in a display region overlapping with theoutside scenery; a memory configured to store selection information tobe used for selecting the objects to be displayed in the display region;and a processor configured to select any of the objects to be displayedin the display region, based on the selection information, and tocontrol display positions of the objects in a depth direction of thedisplay region so that a display position of a selected object isdifferent from a display position of another object to be displayed inthe display region, wherein the selection information includes at leastone setting stored prior to display of the objects, and the processor isconfigured to automatically select any of the objects to be displayed inthe display region based on the at least one setting stored prior todisplay of the objects, and to automatically control the displaypositions of the objects in the depth direction of the display regionbased on the at least one setting, wherein the processor is configuredto change the display position of the selected object in the depthdirection every unit time.
 2. The head-mounted type display deviceaccording to claim 1, wherein the processor is configured to adjust thedisplay positions of the objects in the depth direction by changing aconvergence angle between a right eye and a left eye of the user.
 3. Thehead-mounted type display device according to claim 1, wherein theprocessor is configured to control the display positions of the objectsin the depth direction of the display region and display states of theobjects.
 4. The head-mounted type display device according to claim 1,wherein the processor is configured to change the display position ofthe selected object in the depth direction, and alternately switch adisplay state every unit time between a display state in which theselected object is displayed on a user side relative to another objectin the depth direction and a display state in which the selected objectis displayed on a far side relative to the other object in the depthdirection.
 5. The head-mounted type display device according to claim 1,wherein the processor is configured to change the display position ofthe selected object about an imaginary axis set in a predetermineddirection with the display region as a reference.
 6. The head-mountedtype display device according to claim 5, wherein the processor isconfigured to set the imaginary axis in a direction parallel with avertical direction of the display region, and the processor isconfigured to alternately switch a display state every unit time betweena display state in which one end of the selected object in a directionparallel with a horizontal direction of the display region is displayedon a user side in the depth direction relative to another end in thedirection parallel with the horizontal direction of the display regionand a display state in which the other end is displayed on the user siderelative to the one end in the depth direction.
 7. The head-mounted typedisplay device according to claim 6, wherein the processor is configuredto alternately switch the display state every unit time between adisplay state in which a length of the selected object on the one endside is shorter than a length of the selected object on the other endand a display state in which the length of the selected object on theother end side is shorter than the length of the selected object on theone end side.
 8. The head-mounted type display device according to claim1, wherein, the processor is configured to, in a case where the selectedobject is a character string and the other object includes a characterstring, change the display state of the selected object so that at leastone of a color, boldness, a background color, a font type, and a fontsize of the character string as the selected object and at least acorresponding one of a color, boldness, a background color, a font type,and a font size of the character string of the other object aredifferent from each other.
 9. The head-mounted type display deviceaccording to claim 1, wherein the memory is configured to store controldata defining an operation flow including a plurality of operation stepsfor which an execution order is set in advance, and the processor isconfigured to, in a case where the operation flow is executed, switchwhether or not to execute control of the display positions of theobjects in accordance with a setting of the control data regarding theplurality of operation steps.
 10. The head-mounted type display deviceaccording to claim 1, wherein, the processor is configured to, in a casewhere a plurality of the objects are selected, cause the selectedplurality of objects to be sequentially displayed in the display region.11. The head-mounted type display device according to claim 10, wherein,the processor is configured to, in a case where a first object and asecond object are selected as the plurality of objects, control thedisplay positions of the objects in the display region so that a displaystate is alternately switched between a first display state in which thefirst object is displayed on a user side relative to another object andthe second object is displayed on a far side relative to the otherobject in the depth direction, and a second display state in which thesecond object is displayed on the user side relative to the other objectand the first object is displayed on the far side relative to the otherobject in the depth direction.
 12. The head-mounted type display deviceaccording to claim 1, wherein the processor is configured to control thedisplay position of the selected object in the display region so thatthe selected object does not overlap with another object.
 13. Thehead-mounted type display device according to claim 1, wherein theprocessor is configured to select the object including any of acharacter, an icon, and a frame body surrounding at least one of thecharacter and the icon.
 14. The head-mounted type display deviceaccording to claim 1, wherein the selected object is a first portion ofa first character string, and the other object is a second portion ofthe first character string.
 15. A method of controlling a head-mountedtype display device configured to be mounted on a head of a user andthrough which outside scenery is visible, the method comprising:selecting any of objects to be displayed in a display region overlappingwith the outside scenery, based on selection information to be used forselecting the objects; and controlling display positions of the objectsin a depth direction of the display region so that a display position ofa selected object is different from a display position of another objectto be displayed in the display region, wherein the selection informationincludes at least one setting stored prior to display of the objects,the selecting any of objects includes automatically selecting any of theobjects to be displayed in the display region based on the at least onesetting stored prior to display of the objects, and the controllingdisplay positions of the objects includes automatically controlling thedisplay positions of the objects in the depth direction of the displayregion based on the at least one setting, wherein the controllingdisplay positions includes changing the display position of the selectedobject in the depth direction every unit time.
 16. The method accordingto claim 15, wherein the selected object is a first portion of a firstcharacter string, and the other object is a second portion of the firstcharacter string.